Moving asp.net applications to AWS

S3 Performance considerations

1. http://stackoverflow.com/questions/394198/max-files-per-directory-in-s3
2. http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html

• Use snowball for large amounts of data  https://aws.amazon.com/importexport/
• http://s3tools.org/s3cmd  (command line tools for S3)  http://www.s3express.com/

http://www.s3express.com/help/help.html?vv_auth.html  http://www.s3express.com/help/S3Express.pdf

AWS Setup

1. create  windows 2012 R2 with Sql Server t2 medium
2. In server manager add IIS and .NET 4.5
3. install web-platform install
4. install web-deploy 2.6
5. add design website
6. webpublish

http://martinbuberl.com/blog/deploy-aspnet-mvc-on-amazon-ec2-with-web-deploy/

Completed course on AWS RDS

Big Data

http://www.tableau.com/solutions/sql-server-reporting

http://www.bigsql.org/se/architecture.jsp

http://www.tpostgres.org/se/

Data Science

http://www.wzchen.com/data-science-books

http://www.sec.gov/Archives/edgar/data/1447599/000119312515176980/d875679ds1.htm

http://www.quora.com/What-tools-do-data-scientists-use

http://techcrunch.com/2015/06/15/ibm-pours-researchers-and-resources-into-apache-spark-project/

http://www.slideshare.net/JeffMagnusson/qcon-sf-netflix-hadoop-platform-as-a-service

http://www.huffingtonpost.com/2013/08/01/netflix-profiles_n_3685876.html

splunk with mysql

Aws Big Data

http://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/#json-to-binary-avro

http://blog.cloudera.com/blog/2011/05/three-reasons-why-apache-avro-data-serialization-is-a-good-choice-for-openrtb/

https://www.youtube.com/watch?v=9zpKPivMkS0

http://azure.microsoft.com/en-us/documentation/articles/hdinsight-dotnet-avro-serialization/

http://www.spark-stack.org/getting-started/

http://www.networkworld.com/article/2932313/cloud-computing/how-amazon-s-dynamodb-helped-reinvent-databases.html?page=2

https://snowulf.com/2015/08/05/tutorial-aws-api-gateway-to-lambda-to-dynamodb/

https://aws.amazon.com/blogs/aws/category/amazon-dynamo-db/page/2/

https://aws.amazon.com/blogs/aws/amazon-dynamodb-libraries-mappers-and-mock-implementations-galore/

Library:

http://dynastyjs.com/#

Good Video

Amazon DynamoDB Design Patterns for Ultra-High Performance Apps

https://www.youtube.com/watch?t=5&v=Dh8kp1AcRg0

Big Data on Amazon Web Services

http://www.pluralsight.com/courses/big-data-amazon-web-services

Tableau

http://www.tableau.com/solutions/amazon-redshift

https://www.quora.com/What-is-the-difference-between-storage-S3-and-database-DynamoDB-on-AWS

https://en.wikipedia.org/wiki/Apache_Avro

http://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html#r_COPY_command_examples-copy-from-avro

http://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_FUNCTION.html

http://stackoverflow.com/questions/26100988/how-can-i-import-nested-json-data-into-multiple-connected-redshift-subtables

http://snowplowanalytics.com/blog/2013/11/20/loading-json-data-into-redshift/

RedShift

1. cluster

http://www.slideshare.net/JieLi4/amazon-redshift-at-pinterest

http://snowplowanalytics.com/blog/2013/02/20/transferring-data-from-s3-to-redshift-at-the-command-line/

Kinesis

https://www.youtube.com/watch?v=NIa33ZwFa8E

https://en.wikipedia.org/wiki/Lambda_architecture

https://flume.apache.org/index.html

http://kafka.apache.org/

https://www.elastic.co/products/logstash

https://www.linkedin.com/in/pedrosalgado/

http://www.slideshare.net/JieLi4/amazon-redshift-at-pinterest?related=2

Orchestrating the Cloud : back on Content Store

This month it is back to Content Store. The content Store needs few stories to be wrapped up.

The interesting part is in these stories.

1) AWS Cloudformation :

   I orchestrated the  content store stack using aws cloudformation.  Right from dns(route 53), Elastic        load balancer, Cloud front, S3, ec2, access control,MongoDb replica set, all in < 1000 lines of code. Wow!! what powerful framework.  The one think which puzzles me in this whole equation is which framework is authoritative for running script, is it the cloudinit, cfn-init or puppet . Right now I am using the hybrid approach of loading basic user-data and handing the control to cfn-init.


http://aws.amazon.com/cloudformation/aws-cloudformation-templates/

2) MongoDB Replica set:

  This was another interesting story to setup mongoDB replica set.

         http://www.mongodb.org/display/DOCS/Replica+Sets

3) EveryAuth:

 

  Authentication and Authorization in single api,google hybrid auth :) thats cool.