Services feature end of support

[spectravp]

I received the following email: 

"We noticed you are an admin for a Bitbucket Cloud repository that is still utilizing our original Services feature. We had previously announced that we will end support for this feature and recommended migrating repositories to Bitbucket webhooks."

That's great, but I have a couple hundred repositories. How do you know which one is using the "Services" feature? Do I have to check every repository??

[Leah Rivkin · Accepted solution]

Jeremiah, thanks for this starter script! There are a few optimizations that should fix any insufficient resource issues. For starters, if you use ?role=admin you'll get a smaller subset of all repositories. Also, with BBQL you can restrict the size of the repository response to only include the workspace slug + repository slug. Your new URL should be:

https://bitbucket.org/!api/2.0/repositories?role=admin&pagelen=100&fields=*,values.full_name 

The bigger fix is using our v1.0 API, which is deprecated but can temporarily be queried to check which of your repositories have services that need to be replaced with webhooks. For more information: see our deprecation notice.

https://bitbucket.org/!api/1.0/repositories/${repository.full_name}/services 

The script would then look like this:

(async () => {
let repositoriesPage = 1;

const parseRepositoriesResp = async (resp) => {
    console.log(`Fetched repositories page ${repositoriesPage}`);
    const respJSON = await resp.json();

    if (respJSON.next !== undefined) {
        repositoriesPage++;
        return [
            ...respJSON.values,
            ...(await fetch(respJSON.next).then(parseRepositoriesResp)),
        ];
    }
    return respJSON.values;
};

const parseServicesResp = async (resp) => {
    const services = await resp.json();
    if (services.length > 0) {
        return services.filter(service =>
            service.service.type !== "Issues"
        );
    }
}

console.log('Fetching repositories…');

const repositories = await fetch('https://bitbucket.org/!api/2.0/repositories?role=admin&pagelen=100&fields=*,values.full_name')
.then(parseRepositoriesResp);

console.log('Done fetching repositories');

console.log('Checking for usage of Services…');

const servicesUrls = (
    await Promise.all(repositories.map(async (repository) => {
        const adminHooksUrl = `https://bitbucket.org/!api/1.0/repositories/${repository.full_name}/services`;
        // console.log(`Checking ${repository.full_name}…`);
        const servicesAdminUrl = `https://bitbucket.org/${repository.full_name}/admin/hooks`;
        const repositoryServices = await fetch(adminHooksUrl).then(parseServicesResp);
        if (repositoryServices !== undefined && repositoryServices.length > 0) {
            // console.log(`Repository ${repository.full_name} has services`, repositoryServices)
            return servicesAdminUrl;
        }
    }))
).filter(repo => repo !== undefined);

console.log('Done checking for usage of Services');
console.log('Repositories that use Services:', servicesUrls);
})();

[Jeremiah]

UPDATE: Please see @Leah Rivkin's amazing answer below, which is much more efficient. Thanks Leah!

---

I hacked something together to detect which repositories use Services. Open a JavaScript console on a page on bitbucket.org and run the following:

repositoriesPage = 1; parseRepositoriesResp = async (resp) => { console.log(`Fetched repositories page ${repositoriesPage}`); const respJSON = await resp.json(); if (respJSON.next !== undefined) { repositoriesPage++; return [...respJSON.values, ...(await fetch(respJSON.next).then(parseRepositoriesResp))]; } else { return respJSON.values; } }; domParser = new DOMParser(); console.log('Fetching repositories…'); repositories = await fetch('https://bitbucket.org/!api/2.0/repositories?role=member&pagelen=100').then(parseRepositoriesResp); console.log('Done fetching repositories'); console.log('Checking for usage of Services…'); servicesUrls = (await Promise.all(repositories.map(async (repository) => { const adminHooksUrl = `https://bitbucket.org/${repository.workspace.slug}/${repository.slug}/admin/hooks?iframe=true`; console.log(`Checking ${repository.workspace.slug}/${repository.slug}…`); return await fetch(adminHooksUrl).then(async (resp) => { if (!resp.ok) { return; } const respHTML = domParser.parseFromString(await resp.text(), 'text/html'); if (respHTML.getElementsByClassName('hook').length > 0) { return adminHooksUrl; } }); }))).filter(adminHooksUrl => adminHooksUrl !== undefined); console.log('Done checking for usage of Services'); console.log('Repositories that use Services:', servicesUrls);

This will output an array of URLs for repositories that use Services.

Note that you may see HTTP 403 (Forbidden) errors while it's running; you can ignore these.

[David Spies]

You sir are a steely-eyed missile man!  Thanks for this.

[Blaine Trimmell]

it runs but it IDed repos that do not have Services setup, I will try and debug it but it does find one that has it but some that do not.

[Blaine Trimmell]

I have to update, the script runs and it did id all of them correctly, it is just finding stuff that the BiuBucket UI does not list having Services but when you use the URL returned by Jeremiah's script it does show services on.

Great job Jeremiah

[Chris Birch]

Ok, now that was Brilliant!  Thanks so much for sharing!!!

[Swanthe Lindgren]

Thanks, you are a true JS wizard

[benzhi86]

Nice, thanks!

[Matthias]

Pretty much all of what’s been said about the "helpful" message from Atlassian was what we were discussing in the team this morning…

What even are those Services? And in which of our many repos do we use them??

My colleague then found this thread here and we had a good laugh due to all the comments that stated exactly what we were discussing before.

And then… This script snippet! Ran this in the console and there was the list of (pretty old) repos that use these services. One should think that it should also be possible for Atlassian to generate such a list and inform their users with more details… but yeah.

Thank you very much Jeremiah! Great job.

[Barry Hoofwijk]

Thank you Jeremiah, this worked perfectly. Would indeed have been nice if the 'affected' repos were provided by Atlassian or could be found more easily.

[Mats Rahm]

Hi.

I usually do not run java scripts, so I am a bit at a loss when things go wrong.

We have a huge Bitbucket instance, and when I try to run the script in the Chrome Console, I get an ERR_INSUFFICIENT_RESOURCES error.

How can I increase the resources for the script to run to the end?

[lopezvit]

Thanks for your script. I realized that there are a lot of repositories that only have one service: Issues.

Just to save some time to those that would run into the same circumstance, it is good to know that said service is added as a hook when you add a Private (maybe also Public) Issue tracker (the one that comes with BitBucket, not Jira). It also happens when you do it now.... Which seems crazy, since it is done automatically by Atlassian, and it is using something that it is deprecated.

[Applied Geographics]

Yeah, I think you can probably disregard the repos that only have the Issues service and no other services since the "Services" link doesn't even show up in the admin UI if that's the case. I could be wrong but I updated the script to filter out those repos.

(async () => {
let repositoriesPage = 1;

const parseRepositoriesResp = async (resp) => {
console.log(`Fetched repositories page ${repositoriesPage}`);
const respJSON = await resp.json();
if (respJSON.next !== undefined) {
repositoriesPage++;
return [
...respJSON.values,
...(await fetch(respJSON.next).then(parseRepositoriesResp)),
];
}

return respJSON.values;
};

const domParser = new DOMParser();
console.log('Fetching repositories…');

const repositories = await fetch('https://bitbucket.org/!api/2.0/repositories?role=member&pagelen=100')
.then(parseRepositoriesResp);
console.log('Done fetching repositories');
console.log('Checking for usage of Services…');
const servicesUrls = (
await Promise.all(repositories.map(async (repository) => {
const adminHooksUrl = `https://bitbucket.org/${repository.workspace.slug}/${repository.slug}/admin/hooks?iframe=true`;
console.log('adminHooksUrl:', adminHooksUrl);
console.log(`Checking ${repository.workspace.slug}/${repository.slug}…`);
return fetch(adminHooksUrl)
.then(async (resp) => {
if (!resp.ok) {
return;
}
const respHTML = domParser.parseFromString(await resp.text(), 'text/html');
const services = Array.from(respHTML.getElementsByClassName('hook-name'))
.map((el) => el.innerText && el.innerText.trim())
.filter((issue) => issue !== 'Issues');
if (services.length) {
return adminHooksUrl;
}
});
}))
).filter((adminHooksUrl) => adminHooksUrl !== undefined);
console.log('Done checking for usage of Services');
console.log('Repositories that use Services:', servicesUrls);
})(); 

[Patrick Decat]

The `net::ERR_INSUFFICIENT_RESOURCES` error seems to only be an issue in Chrome, and does not happen in Firefox.

[Mats Rahm]

Thanx. It worked in Firefox.

[Lee Willis]

I'm in the same boat. The email is the most unhelpful communication ever. Either it should contain information about the affected repositories, or there should be an easy way to find out which they are.

[jon_tjs]

It would also be useful if it said what these "Original Services" actually are. I've never used them and nor have any of my current colleagues. Presumably a former colleague must have. Searching for "bitbucket services", unsurprisingly, yields nothing relevent and a lot of noise.

[Chris Birch]

Same with us.  You all stated it perfectly.  Hopefully Atlassian will provide some clarification either on this thread or in a follow-up email. Or if someone else figures out how to see that a repo is at risk, they can post it here.  What a pain to go through several hundred repos though.  We'd have been extremely grateful to get a listing of which of our repos are at risk.

[Chris Birch]

Ok, we were in the same boat, and here's what I've found by poking around.  The email sent has a link to "previously announced", and I see that end-of-life was originally announced for July 1, 2019.  Again, it refers to some "Services" feature.  When I go to a repo in BitBucket and go to Repository settings under "Workflow", I see "Webhooks" but do not see "Services".  But then I went to one of our oldest repos (made a LONG time ago) and, behold, there IS a "Services" option in the Repository Settings under Workflow.  Under "Webhooks" is "Links" and then after that is "Services".  And, sure enough, there are some really old POST services defined.  Probably a left-over of some old integration we no longer use.

Go to "Repository Settings", and IF there is a menu option "Services" under the "Workflow" menu section, there are items there to be deleted or replaced with Webhooks.

But I sure wish there was a way to get a listing of offending repos.

[dmartin-mc]

Found this older announcement - https://bitbucket.org/blog/the-new-bitbucket-webhooks - showing both a screenshot describing what you've found here @Chris Birch as well as a mention of "Services" previously being named "POST and Pull Request POST" hooks.

[seanaty]

> But then I went to one of our oldest repos (made a LONG time ago) and, behold, there IS a "Services" option in the Repository Settings under Workflow.

The feature was soft-deprecated in 2019. Effectively making it such that no one could add a new service and also hiding the "Services" section in the UI unless a repository already had one set.

As you can assume from the email announcement, the were never removed after that soft deprecation and we are doing it now.

[dmartin-mc]

We ended up writing a quick WebDriver script to comb through our repos looking for anything with the "Services" link in the left navigation menu...already had a similar script and was easy to repurpose it. I figured there may have been an API that we could have used but was feeling adventurous.

[jon_tjs]

Many thanks to Jeremiah and Applied Geographics for the script, and Leah at Atlassian too. We have now tracked down and removed the offending services from our repos.

Leah and seanaty, perhaps you could feed back to the appropriate people at Atlassian just how lucky they are to have people like Jeremiah and Applied Geographics as customers, to fill in the gaps in their communications. If you haven't already, of course.

[benzhi86]

It gets even more fuzzy for me when I read this page: https://support.atlassian.com/bitbucket-cloud/docs/view-end-of-support-announcements-for-bitbucket-cloud/?utm_source=alert-email&utm_medium=email&utm_campaign=bb-service-feature_EML-11583&jobid=105241631&subid=1531747445

Do I understand it correctly that pulling and pushing with git from my host machine's terminal will also be suspended? I mean, technically those are Git operations over HTTPS, right?

[seanaty]

No. Relevant thread here: https://community.atlassian.com/t5/Bitbucket-questions/What-does-end-of-support-for-passwords-mean-for-me/qaq-p/1851002

Really concise explanation:

You current Atlassian PW will only work for logging into the web UI. Any other auth (API, git over HTTPS) will require you to create a new app password.

[benzhi86]

Thanks for the explanation!