O'Reilly logo

Apache Spark for Data Science Cookbook by Padma Priya Chitturi

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

NER with IPython over Spark

Apart from POS, one of the most common labeling problems is finding entities in the text. Typically, NER constitutes name, location and organizations. There are NER systems that tag more entities than just these three such as labeling and named entities using the context and other features. There is a lot more research going on in this area of NLP, where people are trying to tag biomedical entities, product entities, and so on.

Getting ready

To step through this recipe, you will need a running Spark cluster either in pseudo distributed mode or in one of the distributed modes, that is, standalone, YARN, or Mesos. Also, have PySpark and Ipython installed on the Linux machine, that is, Ubuntu 14.04. For installing IPython, ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required