Showing posts with label bigData. Show all posts
Showing posts with label bigData. Show all posts
Sunday, September 20, 2020
Thursday, January 16, 2020
Saturday, October 26, 2019
Sunday, July 7, 2019
Sunday, June 9, 2019
Saturday, June 8, 2019
Course : Introduction to Spark SQL and DataFrames
# install python3
install python3
# install spark
https://spark.apache.org/downloads.html
# set environment variables
export SPARK_HOME="/Users/sri/Projects/Playground/spark/"
export PATH=$SPARK_HOME/bin:$PATH
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPRAK_DRIVER_PYTHON_OPTS='notebook'
export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
# install pyspark
pip3 install pyspark
# install jupyter notebook
python3 -m install jupyter
# launch the jupyter notebook
jupyter notebook
https://www.linkedin.com/learning/introduction-to-spark-sql-and-dataframes/load-data-into-dataframes-csv-files
Monday, February 18, 2019
Monday, February 11, 2019
Sunday, February 10, 2019
Tuesday, May 29, 2018
Big Data - Part 7
https://docs.aws.amazon.com/pinpoint/latest/developerguide/integrate-supported-sdks.html
https://docs.aws.amazon.com/mobileanalytics/latest/ug/PutEvents.html
https://www.sumologic.com/blog/devops/kinesis-streams-vs-firehose/
AWS Mobile
https://docs.aws.amazon.com/aws-mobile/latest/developerguide/getting-started.html
https://aws.amazon.com/mobile/
Amazon Mobile Analytics
https://docs.aws.amazon.com/mobileanalytics/latest/ug/welcome.htmlhttps://docs.aws.amazon.com/mobileanalytics/latest/ug/attributes-and-metrics.html
https://docs.aws.amazon.com/mobileanalytics/latest/ug/migrate-autoexport.html
AWS SDK Support for Amazon Pinpoint
https://docs.aws.amazon.com/pinpoint/latest/developerguide/welcome.htmlhttps://docs.aws.amazon.com/pinpoint/latest/developerguide/integrate-supported-sdks.html
https://github.com/aws/aws-sdk-mobile-analytics-js
Streaming Amazon Pinpoint Events to Kinesis
?
https://github.com/aws/aws-amplify
Wednesday, November 29, 2017
Sunday, August 27, 2017
Monday, July 24, 2017
Redshift Spectrum
http://docs.aws.amazon.com/redshift/latest/dg/c-getting-started-using-spectrum-query-s3-data.html
Dabbling with Redshift Spectrum
Amazon Redshift uses massively parallel processing (MPP) to achieve fast execution of complex queries operating on large amounts of data. Redshift Spectrum extends the same principle to query external data, using multiple Redshift Spectrum instances as needed to scan files. Place the files in a separate folder for each table.
Sunday, July 23, 2017
Big Data - Part 6
- Asp .net to parse Query
- Parquet as Storage format
- https://github.com/google/protobuf
https://www.sumologic.com/blog/devops/kinesis-streams-vs-firehose/
https://www.sumologic.com/wp-content/uploads/DemystifyingAmazonKinesis_infographic.pdf
Kafka Vs Kineis
https://blog.insightdatascience.com/ingestion-comparison-kafka-vs-kinesis-4c7f5193a7cd
https://hackernoon.com/processing-real-time-big-data-streams-using-kinesis-lambda-561a029ef305
Sunday, July 2, 2017
Monday, January 2, 2017
Monday, December 26, 2016
AWS Athena - ELB Log Analaysis
Thoughts
Able to get working Athena on our production logs.
No more ELK Stack, Hadoop , Map reduce, Big Query.
Create database apilogs2
CREATE EXTERNAL TABLE IF NOT EXISTS apilogs2.elb_logs_raw_native (
request_timestamp string,
elb_name string,
request_ip string,
request_port int,
backend_ip string,
backend_port int,
request_processing_time double,
backend_processing_time double,
client_response_time double,
elb_response_code string,
backend_response_code string,
received_bytes bigint,
sent_bytes bigint,
request_verb string,
url string,
protocol string,
user_agent string,
ssl_cipher string,
ssl_protocol string )
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
'serialization.format' = '1','input.regex' = '([^ ]*) ([^ ]*) ([^ ]*):([0-9]*) ([^ ]*):([0-9]*) ([.0-9]*) ([.0-9]*) ([.0-9]*) (-|[0-9]*) (-|[0-9]*) ([-0-9]*) ([-0-9]*) \\\"([^ ]*) ([^ ]*) (- |[^ ]*)\\\" (\"[^\"]*\") ([A-Z0-9-]+) ([A-Za-z0-9.-]*)$' )
LOCATION 's3://....';
SELECT * FROM apilogs2.elb_logs_raw_native WHERE elb_response_code = '200' LIMIT 100;
SELECT * FROM apilogs2.elb_logs_raw_native WHERE elb_response_code = '200' and url like '%serialNumber%' LIMIT 100;
SELECT * FROM apilogs2.elb_logs_raw_native WHERE elb_response_code = '200'
and url like '%serialNumber%'
and user_agent like '%ok%' LIMIT 100; --18
Subscribe to:
Posts (Atom)






































