apache spark - Why specifying schema to be DateType / TimestampType will make querying extremely slow? -

-

i'm using spark-csv 1.1.0 , spark 1.5. make schema follows:

private def makeschema(tablecolumns: list[sparksqlfieldconfig]): structtype = {     new structtype(       tablecolumns.map(p => p.columndatatype match {         case fielddatatype.integer => structfield(p.columnname, integertype, nullable = true)         case fielddatatype.decimal => structfield(p.columnname, floattype, nullable = true)         case fielddatatype.string => structfield(p.columnname, stringtype, nullable = true)         case fielddatatype.datetime => structfield(p.columnname, timestamptype, nullable = true)         case fielddatatype.date => structfield(p.columnname, datetype, nullable = true)         case fielddatatype.boolean => structfield(p.columnname, booleantype, nullable = false)         case _ => structfield(p.columnname, stringtype, nullable = true)       }).toarray     )   }

but when there datetype columns, query dataframes slow. (the queries simple groupby(), sum() , on)

with same dataset, after commented 2 lines map date datetype , datetime timestamptype(that is, map them stringtype), queries become faster.

what possible reason this? thank much!

we have found possible answer problem.

when specifying column datetype or timestamptype, spark-csv try parse dates internal formats each line of row, makes parsing progress slower.

from official documentation, seems can specify in option format dates. suppose can make parsing progress faster.


Comments

Post a Comment

Popular posts from this blog

android - Why am I getting the message 'Youractivity.java is not an activity subclass or alias' -

-
Image
i created own android project , cut , pasted code exercise: http://androidexample.com/show_phone_contacts_in_autocomplete_suggestions_-_android_example%20/index.php?view=article_discription&aid=106&aaid=128# when build project no errors or warnings. when run project window error: annoyingly, can't make window smaller, can't click of buttons. cick 'x' @ top, close it. here's project structure: where getting 'autocompletemain' from? (note small c) class named autocompletemain. help. and have activity included in android manifest, in case you're wondering: <?xml version="1.0" encoding="utf-8"?> <manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.androidexample.autocompleteedittext" android:versioncode="1" android:versionname="1.0" > <uses-permission android:name="android.permission.read_phone_state...
Read more

python - How do I create a list index that loops through integers in another list -

-
i have list: sentences = ['the number of adults read @ least 1 novel within past 12 months fell 47%.', 'fiction reading rose 2002 2008.', 'the decline in fiction reading last year occurred among men.', 'women read more fiction.', '50% more.', 'though decreased 10% on last decade.', 'men more read nonfiction.', 'young adults more read fiction.', 'just 54% of americans cracked open book of kind last year.', 'but novels have suffered more nonfiction.'] and have list containing indexes of sequences of sentences in above list contain number. index_groupings = [[0, 1], [4, 5], [8]] i want extract specified sentence sequences in variable "sentences" using indexes in variable "index_groupings" following output: the number of adults read @ least 1 novel within past 12 months fell 47%.fiction reading rose 2002 2008. 50% more.though decreased 10% on last decade. just 54% of a...
Read more

c# - “System.Security.Cryptography.CryptographicException: Keyset does not exist” when reading private key from remote machine -

-
i trying decrypt data using certificate private key. works fine when certificate installed on local machine (i using self signed certificate testing , have private key certificate) when try access private key remote machine using same code, "keyset not exist" exception. i using console application testing, , have made sure id have read permissions on private key on remote server. here sample code using: var store = new x509store(@"\\server1\my", storelocation.localmachine); store.open(openflags.readonly); var result = store.certificates.find(x509findtype.findbysubjectname, "server1.test.com", false); var certificate = result[0]; store.close(); //this succeeds both local , remote server var rsapublic = (rsacryptoserviceprovider)certificate.publickey.key; //this succeeds local, fails remote server var rsaprivate = (rsacryptoserviceprovider)certificate.privatekey; here exception call stack unhandled exception: system.security.cryptography.crypto...
Read more