Showing posts with label bigData. Show all posts
Showing posts with label bigData. Show all posts

Saturday, June 8, 2019

Course : Introduction to Spark SQL and DataFrames




# install python3

install python3

# install spark

https://spark.apache.org/downloads.html

# set environment variables


export SPARK_HOME="/Users/sri/Projects/Playground/spark/"
export PATH=$SPARK_HOME/bin:$PATH
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPRAK_DRIVER_PYTHON_OPTS='notebook'
export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH


# install pyspark

pip3 install pyspark

# install jupyter notebook

python3 -m install jupyter

# launch the jupyter notebook


jupyter notebook





https://www.linkedin.com/learning/introduction-to-spark-sql-and-dataframes/load-data-into-dataframes-csv-files



Tuesday, May 29, 2018

Big Data - Part 7




https://docs.aws.amazon.com/pinpoint/latest/developerguide/integrate-supported-sdks.html
https://docs.aws.amazon.com/mobileanalytics/latest/ug/PutEvents.html



https://www.sumologic.com/blog/devops/kinesis-streams-vs-firehose/


AWS Mobile

https://docs.aws.amazon.com/aws-mobile/latest/developerguide/getting-started.html
https://aws.amazon.com/mobile/


Amazon Mobile Analytics

https://docs.aws.amazon.com/mobileanalytics/latest/ug/welcome.html

https://docs.aws.amazon.com/mobileanalytics/latest/ug/attributes-and-metrics.html
https://docs.aws.amazon.com/mobileanalytics/latest/ug/migrate-autoexport.html





AWS SDK Support for Amazon Pinpoint

https://docs.aws.amazon.com/pinpoint/latest/developerguide/welcome.html
https://docs.aws.amazon.com/pinpoint/latest/developerguide/integrate-supported-sdks.html
https://github.com/aws/aws-sdk-mobile-analytics-js


Streaming Amazon Pinpoint Events to Kinesis




?
https://github.com/aws/aws-amplify




Wednesday, November 29, 2017

Aws Mobile Analytics


Dabbling on Aws Mobile Analytics





Monday, July 24, 2017

Redshift Spectrum



http://docs.aws.amazon.com/redshift/latest/dg/c-getting-started-using-spectrum-query-s3-data.html



Dabbling with Redshift Spectrum






Keep your larger fact tables in Amazon S3 and your smaller dimension tables in Amazon Redshift, as a best practice. If you loaded the sample data in Getting Started with Amazon Redshift, you have a table named EVENT in your database. If not, create the EVENT table by using the following command.


Amazon Redshift uses massively parallel processing (MPP) to achieve fast execution of complex queries operating on large amounts of data. Redshift Spectrum extends the same principle to query external data, using multiple Redshift Spectrum instances as needed to scan files. Place the files in a separate folder for each table.


Sunday, July 2, 2017

Monday, December 26, 2016

AWS Athena - ELB Log Analaysis



Thoughts


Able to get working Athena on our production logs.

No more ELK Stack, Hadoop , Map reduce, Big Query.


Create database apilogs2

CREATE EXTERNAL TABLE IF NOT EXISTS apilogs2.elb_logs_raw_native (
  request_timestamp string,
  elb_name string,
  request_ip string,
  request_port int,
  backend_ip string,
  backend_port int,
  request_processing_time double,
  backend_processing_time double,
  client_response_time double,
  elb_response_code string,
  backend_response_code string,
  received_bytes bigint,
  sent_bytes bigint,
  request_verb string,
  url string,
  protocol string,
  user_agent string,
  ssl_cipher string,
  ssl_protocol string )
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'

WITH SERDEPROPERTIES (
         'serialization.format' = '1','input.regex' = '([^ ]*) ([^ ]*) ([^ ]*):([0-9]*) ([^ ]*):([0-9]*) ([.0-9]*) ([.0-9]*) ([.0-9]*) (-|[0-9]*) (-|[0-9]*) ([-0-9]*) ([-0-9]*) \\\"([^ ]*) ([^ ]*) (- |[^ ]*)\\\" (\"[^\"]*\") ([A-Z0-9-]+) ([A-Za-z0-9.-]*)$' )

LOCATION 's3://....';


SELECT * FROM apilogs2.elb_logs_raw_native WHERE elb_response_code = '200' LIMIT 100;


SELECT * FROM apilogs2.elb_logs_raw_native WHERE elb_response_code = '200' and url like '%serialNumber%' LIMIT 100;


SELECT * FROM apilogs2.elb_logs_raw_native WHERE elb_response_code = '200'
and url like '%serialNumber%'
and user_agent like '%ok%' LIMIT 100; --18