Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

OSError: You seem to have cloned a repository without having git-lfs installed

Jabalnur August 16, 2023

I have pushed a FastAPI program that have large file. The repository is successfully pushed, but when the program was running, there is an error about git-lfs. 


Traceback (most recent call last):
15
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/transformers/modeling_utils.py", line 460, in load_state_dict
16
return torch.load(checkpoint_file, map_location="cpu")
17
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/torch/serialization.py", line 815, in load
18
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
19
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/torch/serialization.py", line 1033, in _legacy_load
20
magic_number = pickle_module.load(f, **pickle_load_args)
21
_pickle.UnpicklingError: invalid load key, 'v'.
22
23
During handling of the above exception, another exception occurred:
24
25
Traceback (most recent call last):
26
File "/root/.local/share/virtualenvs/build-3vGKWv3F/bin/uvicorn", line 8, in <module>
27
sys.exit(main())
28
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
29
return self.main(*args, **kwargs)
30
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/click/core.py", line 1078, in main
31
rv = self.invoke(ctx)
32
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
33
return ctx.invoke(self.callback, **ctx.params)
34
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/click/core.py", line 783, in invoke
35
return __callback(*args, **kwargs)
36
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/main.py", line 416, in main
37
run(
38
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/main.py", line 587, in run
39
server.run()
40
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
41
return asyncio.run(self.serve(sockets=sockets))
42
File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
43
return loop.run_until_complete(main)
44
File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
45
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/server.py", line 68, in serve
46
config.load()
47
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/config.py", line 467, in load
48
self.loaded_app = import_from_string(self.app)
49
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/importer.py", line 21, in import_from_string
50
module = importlib.import_module(module_str)
51
File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
52
return _bootstrap._gcd_import(name[level:], package, level)
53
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
54
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
55
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
56
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
57
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
58
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
59
File "/opt/atlassian/pipelines/agent/build/naturalness_score/api.py", line 6, in <module>
60
from .classifier.model import Model, get_model
61
File "/opt/atlassian/pipelines/agent/build/naturalness_score/classifier/model.py", line 110, in <module>
62
model = Model()
63
File "/opt/atlassian/pipelines/agent/build/naturalness_score/classifier/model.py", line 23, in __init__
64
self.model = DistilBertForMaskedLM.from_pretrained(config["PRE_TRAINED_MODEL"],config=self.config)
65
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2629, in from_pretrained
66
state_dict = load_state_dict(resolved_archive_file)
67
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/transformers/modeling_utils.py", line 465, in load_state_dict
68
raise OSError(
69
OSError: You seem to have cloned a repository without having git-lfs installed. Please install git-lfs and run `git lfs install` followed by `git lfs pull` in the folder you cloned.
70

1 answer

1 vote
Patrik S
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
August 17, 2023

Hello @Jabalnur and welcome to the Community!

Based on the logs you have shared, it seems like you are using Bitbucket Pipelines to build/run your code.

When a Pipeline starts, it will automatically clone the repository to a folder inside the build container, but by default, this clone does not include LFS files.

It seems like the program you are executing in pipelines also needs the files that are stored in the LFS portion of your repository. To instruct pipelines to also clone the LFS during its startup you can include the lfs: true attribute in the bitbucket-pipelines.yml file.

You have the option to specify this at the global level (applies to every step) or at the step level (applies only to the step where the attribute was included) : 

  • Global level: Using lfs to download all LFS files in every step : 
    clone:
      lfs: true
    pipelines:
      default:
        - step:
            name: Clone and download
            script:
              - echo "Clone and download my LFS files!"
  • Step level : Using lfs to selectively download LFS files
    pipelines:
      default:
        - step:
            name: Clone with lfs on
            clone:
              lfs: true
            script:
              - ls -lh large-file.zip # 26M large-file.zip
        - step:
            name: Clone with lfs off
            clone:
              lfs: false
            script:
              - apt-get update && apt-get install -y git-lfs
              # Download only desired files
              - git lfs pull --include=large-file.zip
              - ls -lh large-file.zip # 26M large-file.zip

You can find more details on the usage of LFS attribute in the Pipelines | Git Clone Behaviour | LFS documentation

Let me know in case you have any questions.

Thank you, @Jabalnur !

Patrik S

Jabalnur August 28, 2023

Here is my code, could you check it base on your suggestion before. Please let me know if there is anything wrong or another thing I can add.


image: python:3.10

pipelines:
default:
- parallel:
- step:
name: 'Build and Test'
script:
- echo "Your build and test goes here..."
- step:
name: 'Lint'
script:
- echo "Your linting goes here..."
- step:
name: 'Security scan'
script:
- echo "Your security scan goes here..."

- step:
name: 'Deployment to Staging'
deployment: staging
script:
- echo "Your deployment to staging script goes here..."
# Install pipenv
- pip install pipenv
# Remove any existing Pipfile.lock
- rm Pipfile.lock
# Install dependencies from Pipfile
- pipenv install --deploy

# Start UVicorn server
- echo "Starting UVicorn server..."
- pipenv run uvicorn naturalness_score.api:app --host=0.0.0.0 --port=8000

- step:
name: 'Deployment to Production'
deployment: production
trigger: 'manual'
script:
- echo "Your deployment to production script goes here..."
Jabalnur August 28, 2023

Finally I succeeded in deploying my API to deployment, but how do I use the endpoints? 

Screenshot 2023-08-28 at 23.50.36.jpg

Jabalnur August 28, 2023

my deployment to production is still loading until now, Do you have any solution to this problem? and how to call the API which I use FastAPI? This is my API code:

# from typing import Dict

from fastapi import Depends, FastAPI
from pydantic import BaseModel

from .classifier.model import Model, get_model

app = FastAPI()


class NaturalnessRequest(BaseModel):
     text: str

     class Config:
          orm_mode = True


class NaturalnessMaskResponse(BaseModel):
     text_mask_fill: str


class NaturalnessPerplexityResponse(BaseModel):
     perplexity: float


class NaturalnessRetrainResponse(BaseModel):
     status: str



@app.post("/fill_mask", response_model=NaturalnessMaskResponse)
async def fill_mask(request: NaturalnessRequest, model: Model =                  
Depends(get_model)):
     text_mask_fill = model.fill_mask(request.text)
     return NaturalnessMaskResponse(
          text_mask_fill=text_mask_fill
     )


@app.post("/perplexity", response_model=NaturalnessPerplexityResponse)
async def perplexity(request: NaturalnessRequest, model: Model = Depends(get_model)):
     perplexity = model.perplexity(request.text)
     return NaturalnessPerplexityResponse(
          perplexity=perplexity
     )

@app.post("/retrain", response_model=NaturalnessRetrainResponse)
async def retrain(request: NaturalnessRequest, model: Model = Depends(get_model)):
     status = model.retrain(request.text)
     return NaturalnessRetrainResponse(
          status=status
     )

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
CLOUD
PERMISSIONS LEVEL
Site Admin
TAGS
AUG Leaders

Atlassian Community Events