我已经使用etree.fromstring(x)
从lxml创建了一个通用XMLparser。现在,我必须解析如下的XML:
<row acceptedAnswerId="88156" AnswerCount="6" Body="<p>I\'ve just played a game with my kids that basically boils down to: whoever rolls every number at least once on a 6-sided dice wins.</p> <p>I won,eventually,and the others finished 1-2 turns later. Now I\'m wondering: what is the expectation of the length of the game?</p> <p>I know that the expectation of the number of rolls till you hit a specific number is $\\sum_{n=1}^\\infty n\\frac{1}{6}(\\frac{5}{6})^{n-1}=6$.</p> <p>However,I have two questions:</p> <ol> <li>How many times to you have to roll a six-sided dice until you get every number at least once? </li> <li>Among four independent trials (i.e. with four players),what is the expectation of the <em>maximum</em> number of rolls needed? [note: it\'s maximum,not minimum,because at their age,it\'s more about finishing than about getting there first for my kids]</li> </ol> <p>I can simulate the result,but I wonder how I would go about calculating it analytically.</p> <hr> <p>Here\'s a Monte Carlo simulation in Matlab</p> <pre><code>mx=zeros(1000000,1); for i=1:1000000, %# assume it\'s never going to take us &gt;100 rolls r=randi(6,100,1); %# since R2013a,unique returns the first occurrence %# for earlier versions,take the minimum of x %# and subtract it from the total array length [~,x]=unique(r); mx(i,1)=max(x); end %# make sure we haven\'t violated an assumption assert(~any(mx==100)) %# find the expected value for the coupon collector problem expectationForOneRun = mean(mx) %# find the expected number of rolls as a maximum of four independent players maxExpectationForFourRuns = mean( max( reshape( mx,4,[]),[],1) ) expectationForOneRun = 14.7014 (SEM 0.006) maxExpectationForFourRuns = 21.4815 (SEM 0.01) </code></pre> " CommentCount="5" CreationDate="2013-01-24T02:04:12.570" FavoriteCount="9" Id="48396" LastactivityDate="2014-02-27T16:38:07.013" LastEditDate="2013-01-26T13:53:53.183" LastEditorUserId="198" OwnerUserId="198" PostTypeId="1" Score="23" Tags="<probability><dice>" Title="How often do you have to roll a 6-sided dice to obtain every number at least once?" ViewCount="5585" />',' <row AnswerCount="1" Body="<p>Suppose there are $6$ people in a population. During $2$ weeks $3$ people get the flu. Cases of the flu last $2$ days. Also people will get the flu only once during this period. What is the incidence density of the flu?</p> <p>Would it be $\\frac{3}{84 \\text{person days}}$ since each person is observed for $14$ days?</p> " CommentCount="4" CreationDate="2013-01-24T02:23:13.497" Id="48397" LastactivityDate="2013-04-24T16:58:18.773" OwnerUserId="20010" PostTypeId="1" Score="1" Tags="<epidemiology>" Title="Incidence density" ViewCount="288" />',
我们假设我的目标是提取CommentCount值并进行汇总。当我通过PySpark执行此操作时,当然,这只是数据的一小部分。
我尝试将解析器与.filter()
,reduceByKey
结合使用,但是并没有取得太大的成功。大概我上面提到的lxml解析器返回了一个字典,尽管我无法确认是这种情况。
有人能解释在上述XML中聚合CommentCount值的最佳方法吗?
注意:无法在我的系统上安装Databricks,任何解决方案都不需要这样做。