regex - Named Entity Recognition with Regular Expression: NLTK -



regex - Named Entity Recognition with Regular Expression: NLTK -

i have been playing nltk toolkit. come across problem lot , searched solution online got satisfying answer. putting query here.

many times ner doesn't tag consecutive nnps 1 ne. think editing ner utilize regexptagger can improve ner.

example:

input:

barack obama great person.

output:

tree('s', [tree('person', [('barack', 'nnp')]), tree('organization', [('obama', 'nnp')]), ('is', 'vbz'), ('a', 'dt'), ('great', 'jj'), ('person', 'nn'), ('.', '.')])

where

input:

former vice president dick cheney told conservative radio host laura ingraham "was honored" compared darth vader while in office.

output:

tree('s', [('former', 'jj'), ('vice', 'nnp'), ('president', 'nnp'), tree('ne', [('dick', 'nnp'), ('cheney', 'nnp')]), ('told', 'vbd'), ('conservative', 'jj'), ('radio', 'nn'), ('host', 'nn'), tree('ne', [('laura', 'nnp'), ('ingraham', 'nnp')]), ('that', 'in'), ('he', 'prp'), ('', ''), ('was', 'vbd'), ('honored', 'vbn'), ("''", "''"), ('to', 'to'), ('be', 'vb'), ('compared', 'vbn'), ('to', 'to'), tree('ne', [('darth', 'nnp'), ('vader', 'nnp')]), ('while', 'in'), ('in', 'in'), ('office', 'nn'), ('.', '.')])

here vice/nnp, president/nnp, (dick/nnp, cheney/nnp) , correctly extracted.

so think if nltk.ne_chunk used first , if 2 consecutive trees nnp there high chances both refers 1 entity.

any suggestion appreciated. looking flaws in approach.

thanks.

from nltk import ne_chunk, pos_tag, word_tokenize nltk.tree import tree def get_continuous_chunks(text): chunked = ne_chunk(pos_tag(word_tokenize(text))) prev = none continuous_chunk = [] current_chunk = [] in chunked: if type(i) == tree: current_chunk.append(" ".join([token token, pos in i.leaves()])) elif current_chunk: named_entity = " ".join(current_chunk) if named_entity not in continuous_chunk: continuous_chunk.append(named_entity) current_chunk = [] else: go on homecoming continuous_chunk txt = "barack obama great person." print get_continuous_chunks(txt)

[out]:

['barack obama']

but note if continuous chunk not supposed single ne, combining multiple nes one. can't think of such illustration off head i'm sure happen. if not continuous, script above works fine:

>>> txt = "barack obama hubby of michelle obama." >>> get_continuous_chunks(txt) ['barack obama', 'michelle obama']

regex nlp nltk named-entity-recognition

Comments

Post a Comment

Popular posts from this blog

php - Android app custom user registration and login with cookie using facebook sdk -

php - Android app custom user registration and login with cookie using facebook sdk - i developing android application in users need register , log in using facebook. approximately, flow. open app first time, authenticate facebook, data, send info server create them business relationship in application, start session new business relationship in server, cookie perform later interactions. so far good. what need know wich info have send server, can able identify users when log in seccond time. because need start session in server , send cookie android app farther interactions. i'm using php server side language. i thinking getting getaccesstoken() in android facebook sdk, pass server on business relationship creation , storing in android app. but can't manage how login users on sec time utilize app. any suggestions ? in advance ! to uniquely identify user within app, should create request /me , user's id. guaranteed unique , remain con
Read more

django - Access session in user model .save() -

django - Access session in user model .save() - is possible access current session in user model .save()? pseudo code of want achieve: # users.models.py def save(self, *args, **kwargs): created = true if self.pk: created = false super(abstractuser, self).save(*args, **kwargs) # post-save if created: look_for_invite_in_session_and_register_if_found(self, session) seems wrong in architecture. shouldn't access request in models layer. work request must done in view. can this: user, created = abstractuser.objects.get_or_create(name=name) if created: look_for_invite_in_session_and_register_if_found(user, request.session) django session
Read more

php - .htaccess Multiple Rewrite Rules / Prioritizing -

php - .htaccess Multiple Rewrite Rules / Prioritizing - here current htaccess file: rewriteengine on rewritecond %{http_host} !^www\.website\.com [nc] rewritecond %{http_host} ([^.]+)\.website\.com [nc] rewriterule ^(.*)$ http://www.website.com/gotourl.php?urlid=?%1 [l] rewriterule ^(\w+)$ ./gotourl.php?urlid=$1 [nc,l] rewriterule ^(\w+)/$ ./gotourl.php?urlid=$1 [nc,l] the website in question url shortener i'm working on testing/developing , personal use. above code may not ideal i'm not familiar .htaccess , made via googling , testing. want url shortening aspect: http://website.com/1234 -> http://website.com/gotourl.php?urlid=1234 http://website.com/1234/ -> http://website.com/gotourl.php?urlid=1234 the problem cannot figure out how ignore directories. want specific directories go specific pages: http://website.com/img (or) http://website.com/img/ -> http://website.com/img.php http://website.com/img/1234 (or) http://website.com/
Read more