OSError: You seem to have cloned a repository without having git-lfs installed

[Jabalnur]

I have pushed a FastAPI program that have large file. The repository is successfully pushed, but when the program was running, there is an error about git-lfs. 

Traceback (most recent call last):
15
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/transformers/modeling_utils.py", line 460, in load_state_dict
16
return torch.load(checkpoint_file, map_location="cpu")
17
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/torch/serialization.py", line 815, in load
18
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
19
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/torch/serialization.py", line 1033, in _legacy_load
20
magic_number = pickle_module.load(f, **pickle_load_args)
21
_pickle.UnpicklingError: invalid load key, 'v'.
22
23
During handling of the above exception, another exception occurred:
24
25
Traceback (most recent call last):
26
File "/root/.local/share/virtualenvs/build-3vGKWv3F/bin/uvicorn", line 8, in <module>
27
sys.exit(main())
28
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
29
return self.main(*args, **kwargs)
30
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/click/core.py", line 1078, in main
31
rv = self.invoke(ctx)
32
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
33
return ctx.invoke(self.callback, **ctx.params)
34
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/click/core.py", line 783, in invoke
35
return __callback(*args, **kwargs)
36
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/main.py", line 416, in main
37
run(
38
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/main.py", line 587, in run
39
server.run()
40
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
41
return asyncio.run(self.serve(sockets=sockets))
42
File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
43
return loop.run_until_complete(main)
44
File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
45
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/server.py", line 68, in serve
46
config.load()
47
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/config.py", line 467, in load
48
self.loaded_app = import_from_string(self.app)
49
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/importer.py", line 21, in import_from_string
50
module = importlib.import_module(module_str)
51
File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
52
return _bootstrap._gcd_import(name[level:], package, level)
53
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
54
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
55
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
56
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
57
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
58
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
59
File "/opt/atlassian/pipelines/agent/build/naturalness_score/api.py", line 6, in <module>
60
from .classifier.model import Model, get_model
61
File "/opt/atlassian/pipelines/agent/build/naturalness_score/classifier/model.py", line 110, in <module>
62
model = Model()
63
File "/opt/atlassian/pipelines/agent/build/naturalness_score/classifier/model.py", line 23, in __init__
64
self.model = DistilBertForMaskedLM.from_pretrained(config["PRE_TRAINED_MODEL"],config=self.config)
65
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2629, in from_pretrained
66
state_dict = load_state_dict(resolved_archive_file)
67
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/transformers/modeling_utils.py", line 465, in load_state_dict
68
raise OSError(
69
OSError: You seem to have cloned a repository without having git-lfs installed. Please install git-lfs and run `git lfs install` followed by `git lfs pull` in the folder you cloned.
70

[Patrik S]

Hello @Jabalnur and welcome to the Community!

Based on the logs you have shared, it seems like you are using Bitbucket Pipelines to build/run your code.

When a Pipeline starts, it will automatically clone the repository to a folder inside the build container, but by default, this clone does not include LFS files.

It seems like the program you are executing in pipelines also needs the files that are stored in the LFS portion of your repository. To instruct pipelines to also clone the LFS during its startup you can include the lfs: true attribute in the bitbucket-pipelines.yml file.

You have the option to specify this at the global level (applies to every step) or at the step level (applies only to the step where the attribute was included) : 

• Global level: Using lfs to download all LFS files in every step : 
  

  clone:
    lfs: true
  pipelines:
    default:
      - step:
          name: Clone and download
          script:
            - echo "Clone and download my LFS files!"

• Step level : Using lfs to selectively download LFS files
  

  pipelines:
    default:
      - step:
          name: Clone with lfs on
          clone:
            lfs: true
          script:
            - ls -lh large-file.zip # 26M large-file.zip
      - step:
          name: Clone with lfs off
          clone:
            lfs: false
          script:
            - apt-get update && apt-get install -y git-lfs
            # Download only desired files
            - git lfs pull --include=large-file.zip 
            - ls -lh large-file.zip # 26M large-file.zip

You can find more details on the usage of LFS attribute in the Pipelines | Git Clone Behaviour | LFS documentation

Let me know in case you have any questions.

Thank you, @Jabalnur !

Patrik S

[Jabalnur]

Here is my code, could you check it base on your suggestion before. Please let me know if there is anything wrong or another thing I can add.

image: python:3.10

pipelines:

default:

- parallel:

- step:

name: 'Build and Test'

script:

- echo "Your build and test goes here..."

- step:

name: 'Lint'

script:

- echo "Your linting goes here..."

- step:

name: 'Security scan'

script:

- echo "Your security scan goes here..."

- step:

name: 'Deployment to Staging'

deployment: staging

script:

- echo "Your deployment to staging script goes here..."

# Install pipenv

- pip install pipenv

# Remove any existing Pipfile.lock

- rm Pipfile.lock

# Install dependencies from Pipfile

- pipenv install --deploy

# Start UVicorn server

- echo "Starting UVicorn server..."

- pipenv run uvicorn naturalness_score.api:app --host=0.0.0.0 --port=8000

- step:

name: 'Deployment to Production'

deployment: production

trigger: 'manual'

script:

- echo "Your deployment to production script goes here..."

[Jabalnur]

Finally I succeeded in deploying my API to deployment, but how do I use the endpoints? 

[Jabalnur]

my deployment to production is still loading until now, Do you have any solution to this problem? and how to call the API which I use FastAPI? This is my API code:

# from typing import Dict

from fastapi import Depends, FastAPI

from pydantic import BaseModel

from .classifier.model import Model, get_model

app = FastAPI()

class NaturalnessRequest(BaseModel):

     text: str

     class Config:

          orm_mode = True

class NaturalnessMaskResponse(BaseModel):

     text_mask_fill: str

class NaturalnessPerplexityResponse(BaseModel):

     perplexity: float

class NaturalnessRetrainResponse(BaseModel):

     status: str

@app.post("/fill_mask", response_model=NaturalnessMaskResponse)

async def fill_mask(request: NaturalnessRequest, model: Model =                  

Depends(get_model)):

     text_mask_fill = model.fill_mask(request.text)

     return NaturalnessMaskResponse(

          text_mask_fill=text_mask_fill

     )

@app.post("/perplexity", response_model=NaturalnessPerplexityResponse)

async def perplexity(request: NaturalnessRequest, model: Model = Depends(get_model)):

     perplexity = model.perplexity(request.text)

     return NaturalnessPerplexityResponse(

          perplexity=perplexity

     )

@app.post("/retrain", response_model=NaturalnessRetrainResponse)

async def retrain(request: NaturalnessRequest, model: Model = Depends(get_model)):

     status = model.retrain(request.text)

     return NaturalnessRetrainResponse(

          status=status

     )