{"id":8035,"date":"2020-03-31T18:33:44","date_gmt":"2020-03-31T16:33:44","guid":{"rendered":"https:\/\/blog.via-internet.de\/?p=6344"},"modified":"2020-03-31T18:33:44","modified_gmt":"2020-03-31T16:33:44","slug":"pyspark-cookbook","status":"publish","type":"post","link":"https:\/\/via-internet.de\/blog\/2020\/03\/31\/pyspark-cookbook\/","title":{"rendered":"PySpark | Cookbook"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Websites<\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li>The <a href=\"https:\/\/blaze.pydata.org\/\">Blaze <\/a>Ecosystem (<a href=\"https:\/\/blaze.readthedocs.io\/en\/latest\/index.html\">Blaze<\/a>)<\/li><li><a href=\"https:\/\/dask.readthedocs.io\/en\/latest\/\">Dask<\/a>: Flexible library for parallel computing in Python.<\/li><li><a href=\"https:\/\/datashape.readthedocs.io\/en\/latest\/#\">DataShape<\/a>: Data layout language for array programming.&nbsp;<\/li><li><a href=\"https:\/\/odo.readthedocs.io\/en\/latest\/\">Odo<\/a>: Shapeshifting for your data<br>It efficiently migrates data from the source to the target through a network of conversions.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Reading Textfiles<\/h2>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">from pyspark.sql import SparkSession\n\nspark = SparkSession \\\n    .builder \\\n    .appName(\"Demo\") \\\n    .config(\"spark.demo.config.option\", \"demo-value\") \\\n    .getOrCreate()\n\ndf = spark.read.csv(\"input.csv\",header=True,sep=\"|\");<\/pre>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">from pyspark import SparkContext\nfrom pyspark.sql import SQLContext\nimport pandas as pd\n\nsc = SparkContext('local','example') \nsql_sc = SQLContext(sc)\n\np_df= pd.read_csv('file.csv')  # assuming the file contains a header\n# p_df = pd.read_csv('file.csv', names = ['column 1','column 2']) # if no header\ns_df = sql_sc.createDataFrame(pandas_df)<\/pre>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">sc.textFile(\"file.csv\") \\\n    .map(lambda line: line.split(\",\")) \\\n    .filter(lambda line: len(line)>1) \\\n    .map(lambda line: (line[0],line[1])) \\\n    .collect()<\/pre>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">sc.textFile(\"input.csv\") \\\n    .map(lambda line: line.split(\",\")) \\\n    .filter(lambda line: len(line)&lt;=1) \\\n    .collect()<\/pre>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">spark.read.csv(\n    \"input.csv\", header=True, mode=\"DROPMALFORMED\", schema=schema\n)<\/pre>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">(spark.read\n    .schema(schema)\n    .option(\"header\", \"true\")\n    .option(\"mode\", \"DROPMALFORMED\")\n    .csv(\"input.csv\"))<\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Read CSV file with known structure<\/h4>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">from pyspark.sql.types import StructType, StructField\nfrom pyspark.sql.types import DoubleType, IntegerType, StringType\n\nschema = StructType([\n    StructField(\"A\", IntegerType()),\n    StructField(\"B\", DoubleType()),\n    StructField(\"C\", StringType())\n])\n\n(sqlContext\n    .read\n    .format(\"com.databricks.spark.csv\")\n    .schema(schema)\n    .option(\"header\", \"true\")\n    .option(\"mode\", \"DROPMALFORMED\")\n    .load(\"input.csv\"))<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Websites The Blaze Ecosystem (Blaze) Dask: Flexible library for parallel computing in Python. DataShape: Data layout language for array programming.&nbsp; Odo: Shapeshifting for your dataIt efficiently migrates data from the source to the target through a network of conversions. Reading Textfiles Read CSV file with known structure<\/p>\n","protected":false},"author":1,"featured_media":6348,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[5,16,64],"tags":[],"class_list":["post-8035","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-apache-spark","category-cookbook","category-pyspark"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/posts\/8035","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/comments?post=8035"}],"version-history":[{"count":0,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/posts\/8035\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/menu-items\/6348"}],"wp:attachment":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/media?parent=8035"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/categories?post=8035"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/tags?post=8035"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}