apache spark - kafka to sparkstreaming to HDFS -

-

i using creatdirectstream in order integrate sparkstreaming , kafka. here code used:

val ssc = new streamingcontext(new sparkconf, seconds(10))     val kafkaparams = map("metadata.broker.list" -> "sandbox:6667")     val topics = set("topic1")      val messages = kafkautils.createdirectstream[string, string, stringdecoder, stringdecoder](       ssc, kafkaparams, topics)

now want store messages hdfs. right this?

messages.saveastextfiles("/tmp/spark/messages")

saveastextfiles("/tmp/spark/messages") - persist data in local file system , in case provided folder structure ("/tmp/spark/messages") part of local hdfs show in hdfs directory because saveastextfiles leverages same mapereduce api's write output.

the above work in scenarios spark executors , hdfs on same physical machines in case hdfs directory or url different , not on same machines, executors running, not work.

in case need ensure data persisted in hdfs, practice should provide complete hdfs url. - saveastextfiles("http://<host-name>:9000/tmp/spark/messages")

or can leverage either of following methods: -

  1. dstream.saveasnewapihadoopfiles(<hdfs url location>)
  2. dstream.saveashadoopfiles(<hdfs url location>)

Comments

Post a Comment

Popular posts from this blog

android - Why am I getting the message 'Youractivity.java is not an activity subclass or alias' -

-
Image
i created own android project , cut , pasted code exercise: http://androidexample.com/show_phone_contacts_in_autocomplete_suggestions_-_android_example%20/index.php?view=article_discription&aid=106&aaid=128# when build project no errors or warnings. when run project window error: annoyingly, can't make window smaller, can't click of buttons. cick 'x' @ top, close it. here's project structure: where getting 'autocompletemain' from? (note small c) class named autocompletemain. help. and have activity included in android manifest, in case you're wondering: <?xml version="1.0" encoding="utf-8"?> <manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.androidexample.autocompleteedittext" android:versioncode="1" android:versionname="1.0" > <uses-permission android:name="android.permission.read_phone_state...
Read more

python - How do I create a list index that loops through integers in another list -

-
i have list: sentences = ['the number of adults read @ least 1 novel within past 12 months fell 47%.', 'fiction reading rose 2002 2008.', 'the decline in fiction reading last year occurred among men.', 'women read more fiction.', '50% more.', 'though decreased 10% on last decade.', 'men more read nonfiction.', 'young adults more read fiction.', 'just 54% of americans cracked open book of kind last year.', 'but novels have suffered more nonfiction.'] and have list containing indexes of sequences of sentences in above list contain number. index_groupings = [[0, 1], [4, 5], [8]] i want extract specified sentence sequences in variable "sentences" using indexes in variable "index_groupings" following output: the number of adults read @ least 1 novel within past 12 months fell 47%.fiction reading rose 2002 2008. 50% more.though decreased 10% on last decade. just 54% of a...
Read more

c# - “System.Security.Cryptography.CryptographicException: Keyset does not exist” when reading private key from remote machine -

-
i trying decrypt data using certificate private key. works fine when certificate installed on local machine (i using self signed certificate testing , have private key certificate) when try access private key remote machine using same code, "keyset not exist" exception. i using console application testing, , have made sure id have read permissions on private key on remote server. here sample code using: var store = new x509store(@"\\server1\my", storelocation.localmachine); store.open(openflags.readonly); var result = store.certificates.find(x509findtype.findbysubjectname, "server1.test.com", false); var certificate = result[0]; store.close(); //this succeeds both local , remote server var rsapublic = (rsacryptoserviceprovider)certificate.publickey.key; //this succeeds local, fails remote server var rsaprivate = (rsacryptoserviceprovider)certificate.privatekey; here exception call stack unhandled exception: system.security.cryptography.crypto...
Read more