Persisting a DenseMatrix in Apache Spark -

-

is there recommended/proven efficient format or mechanism persist densematrix in apache spark? or should write file?

i generating densematrix post svd operation , need refer , when user queries come in , hence looked often.

any appreciated.

if densematrix mean org.apache.spark.mllib.linalg.densematrix (v) local data structure , there no spark specific way handle type of objects.

one way handle write serialized object directly file:

val oos = new java.io.objectoutputstream(   new java.io.fileinputstream("/tmp/foo")))  oos.writeobject(svd.v) oos.close()

and read later using fileinputstream , objectinputstream.readobject. can use human readable serialization of choice json:

import net.liftweb.json.{notypehints, serialization} import net.liftweb.json.serialization.{read, write} implicit val formats = serialization.formats(notypehints)  val serialized: string = write(svd.v)  // write file , read if needed ... // deserialize  val deserialized = read[densematrix](serialized)

Comments

Post a Comment

Popular posts from this blog

android - Why am I getting the message 'Youractivity.java is not an activity subclass or alias' -

-
Image
i created own android project , cut , pasted code exercise: http://androidexample.com/show_phone_contacts_in_autocomplete_suggestions_-_android_example%20/index.php?view=article_discription&aid=106&aaid=128# when build project no errors or warnings. when run project window error: annoyingly, can't make window smaller, can't click of buttons. cick 'x' @ top, close it. here's project structure: where getting 'autocompletemain' from? (note small c) class named autocompletemain. help. and have activity included in android manifest, in case you're wondering: <?xml version="1.0" encoding="utf-8"?> <manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.androidexample.autocompleteedittext" android:versioncode="1" android:versionname="1.0" > <uses-permission android:name="android.permission.read_phone_state...
Read more

python - How do I create a list index that loops through integers in another list -

-
i have list: sentences = ['the number of adults read @ least 1 novel within past 12 months fell 47%.', 'fiction reading rose 2002 2008.', 'the decline in fiction reading last year occurred among men.', 'women read more fiction.', '50% more.', 'though decreased 10% on last decade.', 'men more read nonfiction.', 'young adults more read fiction.', 'just 54% of americans cracked open book of kind last year.', 'but novels have suffered more nonfiction.'] and have list containing indexes of sequences of sentences in above list contain number. index_groupings = [[0, 1], [4, 5], [8]] i want extract specified sentence sequences in variable "sentences" using indexes in variable "index_groupings" following output: the number of adults read @ least 1 novel within past 12 months fell 47%.fiction reading rose 2002 2008. 50% more.though decreased 10% on last decade. just 54% of a...
Read more

c# - “System.Security.Cryptography.CryptographicException: Keyset does not exist” when reading private key from remote machine -

-
i trying decrypt data using certificate private key. works fine when certificate installed on local machine (i using self signed certificate testing , have private key certificate) when try access private key remote machine using same code, "keyset not exist" exception. i using console application testing, , have made sure id have read permissions on private key on remote server. here sample code using: var store = new x509store(@"\\server1\my", storelocation.localmachine); store.open(openflags.readonly); var result = store.certificates.find(x509findtype.findbysubjectname, "server1.test.com", false); var certificate = result[0]; store.close(); //this succeeds both local , remote server var rsapublic = (rsacryptoserviceprovider)certificate.publickey.key; //this succeeds local, fails remote server var rsaprivate = (rsacryptoserviceprovider)certificate.privatekey; here exception call stack unhandled exception: system.security.cryptography.crypto...
Read more