cannot import name AddedToken

See original GitHub issue

🐛 Bug

Information

Model I am using (Bert, XLNet …): Albert

Language I am using the model on (English, Chinese …): English

 27 from typing import List, Optional, Sequence, Tuple, Union
 28

—> 29 from tokenizers import AddedToken, Encoding 30 from tokenizers.decoders import Decoder 31 from tokenizers.implementations import BaseTokenizer

ImportError: cannot import name ‘AddedToken’

The problem arises when using:

  • the official example scripts: (give details below)
  • my own modified scripts: (give details below)

The tasks I am working on is:

  • an official GLUE/SQUaD task: (give the name):
  • my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

Expected behavior

Environment info

  • transformers version:
  • Platform:
  • Python version:
  • PyTorch version (GPU?):
  • Tensorflow version (GPU?):
  • Using GPU in script?:
  • Using distributed or parallel set-up in script?:

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:6

github_iconTop GitHub Comments

2reactions
wangxupengcommented, Apr 16, 2020

you need install transformers this way:

git clone https://github.com/huggingface/transformers
cd transformers
pip install .
1reaction
nrjvarshneycommented, Apr 8, 2020

Created another environment from scratch and it got resolved.

Read more comments on GitHub >

github_iconTop Results From Across the Web

ImportError caused by file with the same name in working dir ...
I think I've understood what causes the issue - it is shadowing file with the same name in package transformer (that internally import ......
Read more >
Source code for transformers.tokenization_utils_base
class AddedToken(UserString): # """ AddedToken represents a token to be added to a Tokenizer # An AddedToken can have special options defining the...
Read more >
Hugging Face on Twitter: "@DiogoCortiz Sorry about that. Is ...
Today I am getting the error "cannot import name 'BertWordPieceTokenizer'" when I do "from tokenizers import BertWordPieceTokenizer".
Read more >
tokenizers - PyPI
Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3 most common BPE versions). Extremely fast (both training and ......
Read more >
Source code for paddlenlp.transformers.tokenizer_utils
We can't do it directly within the loop because of how # python iteration works ... whose name ends with `_token`) as attributes...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found